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10 

Description 

The invention relates to an apparatus and a method for/of combining first and second 
image data of an object. An ultrasound (US) detector repeatedly generates the first 

1 5 image data of the object and the second image data of the object are generated 

separately. In the apparatus, means for storing and/or receiving the second image data of 
the object (such as an image data storage and/or an interface) are provided. For example, 
the second image data may have been recorded earlier by a computer tomography (CT), 
a magnetic resonance (MR), a positron emission tomography (PET), an X-ray and/or a 

20 three-dimensional (3D) US imaging device. In particular, any 3D image information can 
be used as the second image data. A combination device combines the first and second 
image data of the object. The combined image data may be displayed in separate areas of 
a screen and/or may be superimposed on a screen. More particularly, the invention may 
be applied in the field of stereographic diagnosis concerning human or animal bodies, but 

2 5 also concerning material research and/or material examination. 

Ultrasound detectors are comparatively easy to handle and are able to deliver image 
information quasi-continuously and, approximately, in real-time. However, in many 
applications, other imaging technologies (such as the ones mentioned above) provide 
better results. Therefore, it has been proposed earlier to combine image information of 

3 0 better quality, which has been recorded earlier, with real-time ultrasound image 

information. 
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In the scientific publication of Pagoulatos et aL: "Interactive 3-D Registration of 
Ultrasound and Magnetic Resonance Images Based on a Magnetic Position Sensor", 
published in IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN 
BIOMEDICINE, VOL. 3, NO. 4, December 1999, describes an apparatus of the type 
5 mentioned above. It is proposed in the publication to use an MR imaging device and to 
register the imaging object relative to the MR imaging device. The term "register" means 
that the geometry of the object and the coordinate system of the MR imaging device are 
brought into a defined relation. The apparatus comprises a position sensor, which is 
firmly coupled to an ultrasound probe. Using the position sensor and due to the feet that 
10 its relative position to the ultrasound probe does not vary, it is possible to track the 

position and the orientation of the ultrasound probe. However, it is necessary to calibrate 
the position sensor relative to the ultrasound probe and to register the imaging object 
. relative to the coordinate system of the ultrasound probe. As a result, the MR imaging 
information and the ultrasound imaging information can be combined. 

15 Using systems of the type described before, it is possible to display MR image 

information of the same orientation and/or scaled in the same manner as the most recent 
US image. In other words: it can be simulated that the MR image is recorded in real- 
time, provided that the image data processing is fast enough. 

However, the various calibration and registration procedures are time-consuming and 
2 0 need to be performed with care in order to obtain good results. For example, the position 
of a set of landmarks on the surface of the object and/or anatomical markers (in case of a 
human or animal body) is to be detected during registration. 

Furthermore, modem US systems enable the user to vary the penetration depth of the 
US image in the object and/or to change the ultrasound probe. As a consequence, the 
2 5 calibration of the position sensor relative to the ultrasound probe and the registration of 
the imaging object relative to the coordinate system of the ultrasound probe are no 
longer valid. 

It is an object of the present invention to provide an apparatus and a method of the type 
indicated above, which allow lessening the effects of the disadvantages mentioned 
30 before, which facilitate the handling by the user and which allow using a greater variety 
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of features of the ultrasound system without loosing time for calibration and/or 
registration. In particular, it is desirable to adjust the settings of the ultrasound system 
and/or to change the ultrasound probe and to continue with the ultrasound imaging 
process without interruption. 

5 A method is provided of combining first and second image data of an object, wherein the 
first image data of the object are repeatedly generated by an ultrasound detector, wherein 
. the second image data of the object are generated separately, wherein the first image data 
are transferred from the ultrasound detector to a combination device and wherein the 
combination device is adapted to combine the first and second image data of the object. 
1 0 Geometry data is transferred in addition to the first image data from the ultrasound 

detector to the combination device, wherein the geometry data comprise information for 
establishing a link between the geometry of the first image data and the geometry of the 
second image data. 

Further, an apparatus is provided for combining first and second image data of an object, 
1 5 the apparatus comprising an ultrasound detector for repeatedly generating the first image 
data of the object; means for storing and/or receiving the second image data of the 
object; a combination device which is adapted to combine the first and second image 
data of the object; and an image data connection from the ultrasound detector to the 
combination device for transferring the first image data. The ultrasound detector is 

2 0 connected to the combination device by a geometry data connection, wherein the 

geometry data connection, the ultrasound detector and the combination device are 
adapted to transfer geometry data additionally to the first image data from the ultrasound 
detector to the combination device and wherein the geometry data comprise information 
for establishing a link between the geometry of the first image data and the geometry of 
25 the second image data. 

The "link" (or, in other words, a processable relation between the first and the second 
image data) allows, for example, determining an overlapping region of a first image 
represented by the first image data and of a second image represented by the second 
image data. According to a preferred application, the combination device continuously 

3 0 displays a first image of the object corresponding to the repeatedly generated first image 
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data and continuously displays a second image of the object coiTesponding to the second 
image data, wherein the orientation and/or scaling of at least a part of the object is 
identical in the first and in the second image. In particular, the second image data may be 
displayed, as if the second image data have repeatedly been recorded instead of or in 
5 addition to the first image data. 

As a consequence of the fact that the geometry data are available to the combination 
device, the effort for re-calibrating the ultrasound detector relative to a tracking sensor 
(if any) and/or for re-registrating the object relative to the coordinate system of the 
ultrasound detector can be reduced or eliminated. 

10 The geometry data may comprise, for example, one or more than one of the following 
type of information: 

a) information concerning at least one spatial dimension of an image unit of the first 
image data, in particular of a pixel (preferably separately for different directions 
of a coordinate system); 

15 b) information concerning an image position of at least a part of an image, which is 
represented by the first image data, relative to a reference point of the ultrasound 
detector or relative to a reference point or reference object in the ultrasound 
image. This information is particularly useful, if a user can adjust a zoom factor 
of the ultrasound image. For example, this information comprises a distance in 

2 0 image units (e.g. pixels). In combination with the scaling information of item a), 

the distance may be defined in cm or another unit of length; 

c) information concerning a detector position of the ultrasound detector relative to a 
position sensor for determining a location and/or an orientation of the ultrasound 
detector. Instead of or in addition to a position sensor, a signal source may be 
2 5 coupled to the ultrasound probe, wherein the signal can be evaluated in order to 

determine the position of the probe. For example, such information may be 
collected once in advance and may be saved individually for each ultrasound 
probe, which can be connected to the ultrasound system/device. In this case, it is 
sufficient during operation to transfer simply an identification signal, which 
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enables to identify the probe that is used. The combination device can select the 
respective geometry information using the identification information. In a specific 
embodiment, the information concerning the relative position, which is 
transferred or saved, may be a calibration matrix; 

5 d) information concerning an orientation of the ultrasound image relative to a 

reference point or a reference object of the ultrasound detector (in particular an 
ultrasound probe of the detector). For example, this information may comprise 
the orientation of at least one axis of a coordinate system of the ultrasound 
image; and 

10 e) information concerning a region or an area, which is actually covered by an 

ultrasound image that is represented by the first image data. 

Preferably, all of these types of information are transferred from the ultrasound detector 
to the combination device. 

According to a preferred embodiment at least a part of the geometry data is repeatedly 
1 5 transferred to the combination device, in particular every time when the first image data 
are generated and/or when the first image data are transferred to the ultrasound detector. 
The geometry data may be transferred on request and/or without request from the 
combination device and/or from another part or unit (for example of a central control 
unit) of the apparatus. 

2 0 There are further possibilities to use the additional data connection between the 

ultrasound detector and the combination device, or to use a further data connection. For 
example, a mode and/or a user setting of the ultrasound detector may be transferred from 
the ultrasound detector to the combination device, on request or without request of the 
combination device or another unit of the apparatus. E.g. a colour of the ultrasound 

2 5 image to be used in displaying the ultrasound image, a repetition frequency of the 

ultrasound image generation (for example in order to give an indication to the user, if the 
frequency is too low) and/or information representing the object may be transferred. 



If the ultrasound detector comprises a control unit for controlling an image data 
generation of the ultrasound detector, the control unit may be adapted to generate at 
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least a part of the geometry data. For example, the control unit can adjust a penetration 
depth of the ultrasound image, using a velocity value of the ultrasound waves in the 
object, by setting a time limit for detection of US echo signals. In this case, the control 
unit can calculate the penetration depth and can transfer information about the 
penetration depth to the combination device. Further, the width of an image recording 
area of an ultrasound probe may be available to the control unit for control purposes and 
the control unit can transfer this information to the combination device. 

In a preferred embodiment of the apparatus the image data connection is adapted to 
directly transfer the first image data in a digital format from the ultrasound detector to 
the combination device. This saves time for transferring the first image data and reduces 
costs and effort of the apparatus. Furthermore, the ultrasound detector, the combination 
device and (optionally) further parts or units of an imaging system may be integrated in 
one and the same device. For example, several or all of the units of such a device may be 
connected to a data bus system for transferring data. 

Generally, the image data connection and/or the geometry data connection may be 
realised by a data bus (e.g. USB or FireWire, IEEE 1394) and/or may be part of a data 
network. Preferably, the embodiment of the connection or connections shall allow 
displaying a first image (represented by the fust image data) and a second image 
(represented by the second image data) approximately in real-time of the generation of 
the first image data, for example within less than 100 ms after the generation. Ultrasound 
is particularly useful in order to generate the first image data, since the generation 
process can be performed with high repetition frequency (quasi-continuously). The 
corresponding data connection and the following data processing should not delay the 
displaying in way, which can be noticed by the user. 

In the following, examples and possible further features of the invention are described by 
way of reference to the accompanied drawing. The embodiment shown in the figures 1 to 
3 and 5 represent the currently known best mode of the invention. However, the 
invention is not limited to the features described in the following description. The figures 
of the drawing schematically show: 
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Fig. 1 an arrangement 2 comprising an apparatus for combining ultrasound 

image data with a second type of data, e.g. CT image data; 

Fig- 2 a more detailed view of the ultrasound detector shown in Fig. 1; 

Fig, 3 a first example of an arrangement of components, which contribute to a 

5 processing of the ultrasound image data; 

Fig. 4 a second example of an arrangement of components, which contribute to 

a processing of the ultrasound image data; 

Fig. 5 a flow-chart of processing image data. 



1 0 Images of an object 3 (shown in Fig. 1) are to be displayed on a screen 6. An ultrasound 
detector 1 generates first image data of the object 3 and transfers the first image data to a 
combination device 5 via an image data connection 1 0. The combination device 5 
comprises a data storage 4. which contains second image data that have previously been 
generated by a separate device (not shown in Fig. 1). The combination device 5 is 

1 5 adapted to combine the first and second image data and to display them on a screen 6, 

which is connected to the combination device 5. For example, the first and second image 
data may be displayed separately on a split screen or may be superimposed. In any case, 
it is preferred that a first image, which is generated using the first image data, and a 
second image, which is generated using the second image data, precisely show at least 

2 0 partially the same area or region of the object 3 in the same orientation (angle of view) 
and scaling (dimensions). 

The ultrasound detector 1 and the combination device 5 are connected to each other by 
an additional data connection 12 for transferring geometry data from the ultrasound 
detector 1 to the combination device 5. In particular, the geometry data connection 12 
25 may be connected (as shown in Fig. 2) to a control unit 14 of the ultrasound detector 1. 

In practice, the data connections 10, 12 may be realised by separate data connection links 
or by the same data connection link. For example, a "link" may comprise a connection 
line, a plurality of connection lines and/or a digital data bus or bus system. 
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An ultrasound probe 16 (Fig. 2) of the ultrasound detector 1 is firmly coupled to a 
position sensor 1 8 of a tracking system. The determination of the orientation and the 
location of such a position sensor and, thereby, of the ultrasound probe is known in the 
art (see the above-cited publication of Pagoulatos et al.). For example, magnetic and/or 
5 optical (e.g. infrared) signals may be used by the tracking system. The position sensor 
18 is connected to a tracking system control unit 8 and the control unit 8 is connected to 
the combination device 5. During operation of the arrangement 2, the control unit 8 
repeatedly or quasi-continuously transfers information concerning the position and 
concerning the orientation of the ultrasound probe 16 to the combination unit 5. 
1 0 Alternatively, this information may be transferred from the US detector to the 

combination device. I.e. this information might be at least partially included in the 
geometry data, which are transferred. 

As shown in Fig. 2, the ultrasound device 1 may, for example, comprise an ultrasound 
probe 16, which is connected to the ultrasound control unit 14 via a flexible cord 17 for 
1 5 transferring echo signals to the control unit 14. On the other hand, the control unit 14 
transfers control signals to the ultrasound probe via the cord 17. Also, it is possible that 
at least a part of the geometry information is transferred from the ultrasound probe 1 6 to 
the control unit 14 and/or that at least a part of the geometry information generated by 
the control unit 14 is based on and/or derived from information, which is transferred 
2 0 from the ultrasound probe 16 to the control unit 14. For example, the ultrasound probe 
16 may be replaced and, therefore, transfers information concerning its identity to the 
combination device 5. In the combination device 5, in an additional unit of the 
arrangement 2, and/or in the combination device 5, information concerning the relative 
position and/or orientation of the specific ultrasound probe 16 relative to the position 

2 5 sensor 1 8 may be saved. For example, the ultrasound probe 16 may comprise a clip for 

attaching the position sensor 1 8. Therefore, it is possible to precisely position and 
orientate the position sensor 18 relative to the ultrasound probe 16 and to determine as 
well as to save the respective geometry data in advance. After replacement of the 
ultrasound probe 1 6, the identity information and the saved geometry information can be 

3 o combined. As a consequence, it is not necessary to re-calibrate the arrangement 

consisting of the ultrasound detector 1 and the position sensor 18. 
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An input unit 20 is connected to the ultrasound control unit 14, for example for inputting 
settings of the ultrasound detector, such as a penetration depth or range of the 
ultrasound image. Further, the user may change the orientation of the ultrasound image 
via the input unit 20. 

5 According to a preferred embodiment of the invention, the ultrasound image data are 
generated from analog signals in a unit 22 of the ultrasound detector. The unit 22 may, 
for example, be controlled by the control unit 14 (as shown in the figure), be a part of the 
control unit 14 or be realised by a combination of the ultrasound probe 16 and the 
control unit 14. The unit 22 is connected to an A/D-converter 24, which converts the 

1 0 analog image signal to a digital image signal. The A/D-converter 24 is connected to an 
image data processing unit 26, which is also connected to the control unit 14 or to an ' 
alternative unit of the ultrasound detector that transfers geometry information to the 
image data processing unit 26. Thus, the image data processing unit 26 can combine the 
geometry information with the image data and/or can transferred both information/data 

15 to a further unit (not shown in figure 3). For example, the image data processing unit 26 
may be identical to or part of the combination device 5. In this case, the image data 
processing unit 26 also combines the first and second image information and the unit 26 
may be connected to a screen 6 (as shown in the figure). 

The embodiment of figure 3 is of particular advantage, if the ultrasound detector and the 
2 0 combination device are integrated in one device. 

In an alternative arrangement, the ultrasound detector may directly generate digital 
image data, without conversion from analog to digital. 

An alternative arrangement is shown in figure 4. The same reference numeral denote 
units or devices having the same or a corresponding function. A D/A-converter 23 
2 5 converts the digital image signal received via a connection from the unit 22 or from a 
corresponding unit to an analog image signal, for example to a signal in standard video 
format. An A/D-converter 25 that may be part of the combination device (e.g. a video 
capture card) is connected to the unit 23. The unit 25 converts the analog signal to a 
digital signal, e.g. in pixel format. Then, the unit 26 processes the digital signal. 
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An exaropie of processing and combining utaasoond image information is described in 
the following with reference to figure 5. 

In step SI, geometry information concerning the scaling of the ultrasound image 
information are combined with the ultrasound image information. For example, the 
ultrasound image data contains information concerning the intensity of image units (e.g. 
pixels) before step S 1 , but no information concerning the dimension of the image umts. 
The dimensional information is added by performing step SI. 

In step S2, calibration information concerning the relative position of the ultrasound 
detector to the position sensor is added and/or combined with the information resultmg 
from step SI. The calibration information may comprise as well information conceding 
the orientation of the ultrasound image, which may be changed by the user. As a result, rt 
is now possible to identify the orientation and location of specific areas of the ultrasound 
image. 

In step S3, the geometry information concerning the registration of the object withinthe 
1 5 coordinate system of the ultrasound system or the tracking system is added. 

In step S4, the ultrasound image data resulting from step S3 can be combined with the 
second image data. 

■ In practice, at least some of the processing operations of steps SI to S4 may be realised 
by performing a polygon scanline method, which is described for example in the book 
"Computer Graphics. Principles and Practice." by James D. Foley, Andries VanDam, 
Steven K. Feiner, published by Addison-Wesley, Boston, U.S.A The combination 
device 5 shown in figure 1 may perform the data processing, for example. 
For example, it is first calculated whether an overlapping region of the respective images 
exists. If this is the case, the corresponding overlapping region of the second image data 
is read out from the data storage and is prepared for displaying (step S5). The 
corresponding data of the second image data are processed before displaying so that the 
second image, which is based on the second image data, can be displayed in the same 
orientation and scaling as the ultrasound image. 
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The procedure described before is preferred, if the ultrasound image is two-dimensional. 
For 3D-ultrasound image information it is as well possible, to choose the orientation (i.e. 
the angle of the view) of the ultrasound image to be shown, in particular to adapt the 
orientation according to other criteria. In any case, it is possible to adapt the scaling of 
5 the ultrasound image before displaying. Instead of, or additionally to displaying, the 
combined first and second image information may be processed in another way. For 
example, they can be stored and/or evaluated in a different manner. 
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Patent Claims 

An apparatus (2) for combining first and second image data of an object (3), 
comprising 



5 . an ultrasound detector (1) for repeatedly generating the first image 

data of the object (3); 
. means (4) for storing and/or receiving the second image data of the 
object (3), for example image data generated by a computer 
tomography, a magnetic resonance, a positron emission tomography 
1Q or an X-ray imaging device; 

. a combination device (5) which is adapted to combine the first and 

second image data of the object (3); and 
. an image data connection (10) from the ultrasound detector (1) to 
the combination device (5) for transferring the first image data; 
15 characterised in that 

the ultrasound detector (1) is connected to the combination device (5) by a 
geometry data connection (12), wherein the geometry data connection (12), 
the ultrasound detector (1) and the combination device (5) are adapted to 
transfer geometry data additionally to the first image data from the ultrasound 
detector (1) to the combination device (5) and wherein the geometry data 
comprise information for establishing a link between the geometry of the first 
image data and the geometry of the second image data. 



The apparatus of claim 1 , wherein the image data connection is adapted to 
directly transfer the first image data in a digital format from the ultrasound 
detector (1) to the combination device (5). 

The apparatus of claim 1 or 2, wherein the ultrasound detector (1) comprises a 
control unit (14) for controlling an image data generation of the ultrasound 
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detector (1) and wherein the control unit (14) is adapted to generate at least a 
part of the geometry data. 

4. A method of combining first and second image data of an object (3), wherein 

5 the first image data of the object (3) are repeatedly generated by an ultrasound 

detector (1), wherein the second image data of the object (3) are generated 
separately, for example the second image data have been recorded earlier by a 
computer tomography, a magnetic resonance, a positron emission tomography 
or an X-ray imaging device, wherein the first image data are transferred from 
1 o the ultrasound detector (1 ) to a combination device (5) and wherein the 

combination device (5) is adapted to combine the first and second image data 
of the object (3), 
characterised by 

transferring geometry data in addition to the first image data from the 
15 ultrasound detector (1) to the combination device (5), wherein the geometry 

data comprise information for establishing a link between the geometry of the 
first image data and the geometry of the second image data. 

The method of claim 4, wherein the geometry data comprise information 
concerning at least one spatial dimension of an image unit of the first image 
data, in particular of a pixel. 



5. 

20 



6. The method of claim 4 or 5, wherein the geometry data comprise information 
concerning an image position of at least a part of an image, which is 

2 5 represented by the first image data, relative to a reference point of the 

ultrasound detector (1) or relative to a reference point or reference object in 

t 

the ultrasound image. 

7. The method of one of claims 4 to 6, wherein the geometry data comprise 

3 0 information concerning a detector position of the ultrasound detector (1) 

relative to a position sensor (1 8) and/or relative to a signal source, for 
determining a location and/or an orientation of the ultrasound detector (1). 
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g The method of one of claims 4 to 7, wherein the geometry data comprise 

information concerning an orientation of the ultrasound image relative to a 
reference point or a reference object of the ultrasound detector (1). 

9 The method of one of claims 4 to 8, wherein the geometry data are generated 

by and/or transferred from a control unit (14) of the ultrasound detector (1) 
and wherein the control unit (14) is adapted to control an image data 
generation of the ultrasound detector (1). 



10 

10. 



11. 



The method of one of claims 4 to 9, wherein the combination device (5) 
continuously displays a first image of the object (3) corresponding to the 
repeatedly generated first image data and continuously displays a second 
image of the object (3) corresponding to the second image data, wherein the 
15 orientation and/or scaling of at least a part of the object (3) is identical in the 

first and in the second image. 



20 



Themethodofoneofclaims4to 10, wherein at least apart of the geometry 
data is repeatedly transferred to the combination device (5), in particular every 
time when the first image data are generated. 



11 Apr 03 22:37 



Laube/Brunotte 



+49 (0)30 8471 6949 p. 18 




11 Apr 03 22:38 



Laube/Brunotte 



+49 (0)30 8471 6949 p. 19 




11 Hpr 03 22:38 



Laube/Brunotte 



+ 49 (0)30 8471 6949 p. 20 



3/3 



SI 






S2 






S3 




f 


S4 






S5 



Fig. 5 



+49 (0)30 8471 6949 p. 21 

11 Rpr 03 22:38 Laube/Brunotte 



11.04.03 

15 

MEDPT03027 



Applicants: 

MedCom Gesellschaft fur medizinische Bildverarbeitung mbH 
R^deturmstrasse 12, D-64283 Darmstadt, GERMANY and 
Esaote S.p.A., via Ruffino Aliora, 32 
15033 Casale Monferrato (AL), ITALY 



10 



15 



20 



Apparatus ,„d Mc.hod for Cmbming Firs, and Second Image »•<• •'«' 



Abstract 



The invention relates to . merhod of combining fir* and second taage data of an obje« 
(3) whereinrta firs, taage data of .he object (3) are repeatedly generated by an 
utoS o«nd de.ec.or (1), wheremthe secondtaage data of the object (3) are generated 
separate.,, for example ft. second taage data have been recorded earUer by a computer 
tomography, a magnetic resonance or a position emission tomography imaging dev.ce, 
wherein .he firs, taage data are transferred from .he ultrasound de.ec.or (I) to a 
combination device (5) andwher.in.he combination device (5) is adapted to combme 
the first and second image data of the object (3). 



Geometry data is transferred in addition to the first image data from the ultrasound 
detector (1) to the combination device (5), wherein the geometry data compnse 
25 infonnationforestab^^ 

geometry of the second image data. 



(Fig. 1) 
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